world onto an image plane, a two-dimensional space. Rectification is the process of reprojecting 
from the image plane onto another plane, which is not necessarily parallel to the image plane. 
When rectifying the image onto an ordinary perspective plane, the rectification can be 
accomplished without knowledge of the three-dimensional locations of the points of the 
underlying three-dimensional world. 

[67] In block 23, an edge operation is applied to the rectified image from block 22 to 
obtain an edge image. The edge operation determines edges of features in the rectified image, 
and the edge image can be regarded as a line-based rendering of the rectified image. Examples 
of an edge operation include Sobel and other gradient operators. 

[68] In block 24, the edge image from block 23 is edited to obtain a modified image. 
The image can be edited in a number of ways as discussed further below with respect to Figure 
4. The image can be edited, for example, by using Adobe PhotoShop. By editing the edge 
image of a camera-motion layer, the entire video sequence relying on the camera-motion layer is 
modified. With the invention, only the camera-motion layer is edited, and the corresponding 
sequence of frames in the composite modified video sequence is affected. This is a powerful 
advantage over the conventional techniques, which require every frame of the video sequence to 
be edited. 

[69] In block 25, the modified image is rectified from the ordinary perspective plane to 
the original image plane. 

[70] In block 26, the modified image rectified in block 25 is converted to obtain a 
modified camera-motion layer. The conversion in block 26 is the reciprocal of the conversion in 
block 21. 
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[71] Figure 4 illustrates various aspects of editing a camera-motion layer in block 13 
of Figure 2 and in block 24 of Figure 3. Figure 4 illustrates various editing possibilities with the 
invention. One or more camera-motion layers can be edited using one or more of the 
possibilities illustrated in Figure 4. The possibilities in Figure 4 are generally described as being 
5 applicable to camera-motion layers because they are applicable to editing both original camera- 
motion layers and modified camera-motion layers. As discussed above, a single change to one 
camera-motion layer affects the entire video sequence of multiple frames relying on the camera- 
Cf motion layer, and the single change does not need to be made laboriously to each frame of the 
original video sequence. 

rft) [72] If a modification discussed herein is saved as an additional camera-motion layer, 

jfy leaving the original camera-motion layer untouched, the additional camera-motion layer can 
N 8 have various effects parameters associated with it, such as an on/off time, a level of opaqueness, 
J: and/or a fade-in/fade-out parameter. 

f* s [73] The editing possibilities depicted in Figure 4 can be accomplished using 

15 conventional software. For example, images can be edited using Photoshop by Adobe Systems 
Incorporated of San Jose, California. As another example, video can be edited using Premiere by 
Adobe Systems Incorporated of San Jose, California. As a further example, three-dimensional 
animation can be performed using 3D Studio MAX by Discreet of Montreal, Quebec, Canada. 
[74] In block 31, an object is inserted in a camera-motion layer. The edges of the 
20 inserted object are preferably blended so that it is difficult to determine that the object is an 

artificial insertion. As an example, a machine for dispensing canned soda can be inserted into a 
video scene of a hotel lobby. In addition, the carpet next to the soda machine can be changed to 
account for the fluorescent light from the soda machine. Further, using on/off times, the soda 
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machine can be seen to appear and then disappear from the hotel lobby during the composite 
modified video sequence. 

[75] In block 32, an object is deleted from a camera-motion layer. One or more 
portions of the static background of the camera-motion can be removed (e.g., "airbrushed out") 
such that it is difficult to determine the portion of the background that has been modified. As an 
example, a door can be deleted from the wall in a video scene of a hotel lobby. In place of the 
door, the pattern of the wall can be repeated. 

[76] In block 33, an object is changed in a camera-motion layer. Changing an object 
includes aspects of inserting an object and deleting an object to make a change to the camera- 
motion layer. As an example, a potted plant in a video scene of a hotel lobby can be moved from 
one location to another. Further, using fade-in/fade-out parameters, the potted plant can appear 
in the composite modified video sequence to vanish slowly from one location and to reappear 
slowly in another location. 

[77] In block 34, the camera-motion layer is entirely replaced with another camera- 
motion layer. The replacing camera-motion layer can be obtained from a video sequence or can 
be an image. Examples of such an image include: a computer-generated image; a computer- 
aided drawing; a digital photograph; a scanned image; and a frame from a video sequence. As an 
example, the guests checking into one hotel lobby in the original video sequence can be viewed 
in the composite modified video sequence to be checking into an entirely different hotel lobby or 
into a cartoon hotel lobby. 

[78] In block 35, a video sequence is added to a camera-motion layer. A video 
window is inserted in the camera-motion layer that maintains the motion characteristics of the 
camera via the camera motion parameters of the decomposed original video sequence. This is an 
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